Object based image retrieval

ABSTRACT

A computer-aided image comparison, evaluation and retrieval system compares objects and object clusters, or images. User controlled or automatic filtering to enhance object features may be performed prior to object definition/detection. The query image may be substantially continuously displayed during the image filtering and object definition processes. Scoring to suspected biological, medical, chemical, physical or clinical condition may be performed based on retrieved objects or images and their relative similarities to the unknown.

This application is a continuation of U.S. Ser. No. 11/007,062, filedDec. 7, 2004, now U.S. Pat. No. 7,483,919 which is acontinuation-in-part of U.S. application Ser. No. 09/370,366 filed Aug.9, 1999, now U.S. Pat. No. 6,941,323. Both of these applications arehereby incorporated by reference in their entireties.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to management of an image database and theretrieval of images therefrom.

2. Description of the Related Technology

The ability to search image databases and retrieve images therefrom withdesired features or characteristics is important in many differentenvironments. However, as a collection of images to be searched grows insize, the ability to search and evaluate the collection manually forimages having the desired features becomes increasingly limited. It canbe appreciated that huge image databases of thousands or even millionsof images have been created which are essentially impossible to searchand evaluate manually.

Several approaches have been used to automate the image search process.In some cases, images are digitized and stored in a database inassociation with one or more keywords which describe their content orcharacter. Such a database can be searched linguistically for particularkeywords, and images which are associated with these keywords areretrieved in response.

In a more recently developed alternative method, one or more “queryimages” are utilized, and images from the database which are in somesense similar to the query image are located and retrieved. In thesesystems, the pixel values of the query image and the images in thedatabase are processed to produce a set of parameters indicative ofcolor distribution, pixel intensity variation across the image, as wellas other characteristics of the image as a whole. These parameters arecalculated using various image filtering and processing techniques so asto produce a vector of feature parameters which is indicative of theimage itself. The comparison process involves comparing feature vectorsfrom images in the database with a query feature vector, and images fromthe database having similar feature vectors are retrieved. A system ofthis nature is described in U.S. Pat. No. 5,644,765 to Shimura et al.,the disclosure of which is hereby incorporated by reference in itsentirety.

The above described systems have several limitations. The most seriousdrawback for both cases is that image content is inadequately defined,which impacts both system recall and precision. Recall is the proportionof relevant images in the database that are retrieved, and precision isthe proportion of retrieved documents that are actually relevant. Thesetwo measures may be traded off one for the other, and the goal of imageretrieval is to maximize them both.

SUMMARY

The invention comprises methods and systems for processing, comparingand retrieving images. In one embodiment, the invention comprises amethod of identifying similarities between first and second imagedstructures present in one or more digital images. The method includesprocessing one or more digital images so as to define at least first andsecond objects, assigning a first object characterization parameter setto the first object, and assigning a second object characterizationparameter set to the second object. The method further comprisescalculating a similarity index based on the first objectcharacterization parameter set and the second object characterizationparameter set.

Scoring to suspected biological, medical, chemical, physical or clinicalcondition may be performed based on retrieved objects or images andtheir relative similarities to the unknown.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of a method of image retrieval in one embodimentof the invention.

FIG. 2 is a block diagram of an image retrieval system according to theinvention which may be utilized to carry out the method of FIG. 1.

FIG. 3 is a conceptual schematic of parameter sets associated withobjects segmented from an image which may be created by the objectparameterzation module of FIG. 2.

FIG. 4 is a flowchart of one embodiment of an object parameterizationprocess which may be implemented in the object parameterization moduleof FIG. 2.

FIG. 5 is a screen display of user configured look up table filterfunctions according to one embodiment of the invention and which may begenerated by the system of FIG. 2.

FIG. 6 is a screen display of user configured sharpening filterfunctions according to one embodiment of the invention and which may begenerated by the system of FIG. 2.

FIG. 7 is a screen display of user configured general and edgeenhancement filter functions according to one embodiment of theinvention and which may be generated by the system of FIG. 2.

FIG. 8 is a screen display of user configured object definitionaccording to one embodiment of the invention and which may be generatedby the system of FIG. 2.

FIG. 9 is a screen display of user configured object searching andcomparison according to one embodiment of the invention and which may begenerated by the system of FIG. 2.

FIG. 10 is a screen display of user configured object searching,comparison and scoring similarity according to one embodiment of theinvention and which may be generated by the system of FIG. 2.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention will now be described with reference to theaccompanying Figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limited or restrictive manner,simply because it is being utilized in conjunction with a detaileddescription of certain specific embodiments of the invention.Furthermore, embodiments of the invention may include several novelfeatures, no single one of which is solely responsible for its desirableattributes or which is essential to practicing the inventions hereindescribed.

In many imaging applications, a user of the system wishes to find imagesin an image database which contain a certain defined object. In somecases, the search may be for images containing a chair, sunset,mountain, or the like. This may be the case in the advertising orcommercial arts fields, for example, where large searchable image filesare kept for use in the production of artwork, posters, etc. Otherapplications may also benefit from a robust object recognition system.In the health care environment, images such as X-ray films, CAT scans,ultrasound and MRI images generally contain specific internal objects ofinterest such as blocked blood vessels, tumors, malignant or benigngrowth and other structures. In many cases, diagnosis and treatmentwould be facilitated if a physician could search and evaluate an imagedatabase for similar structures found in other patients so as to obtainvaluable information regarding diagnosis, treatment, and outcome forother patients showing similar objects under X-ray or MRI imaging. Asyet another example, geographical and geological surveying, mapping, andother forms of reconnaissance (including military targeting systems)would also be facilitated by such a system. Structures in aerial andsatellite photographs could more easily be correlated to specificphysical objects if specific ambiguous structures in the image could becross referenced to structures found in other aerial or satellitephotographs. To address this need, embodiments of the invention allow auser to focus image database searching and evaluation on objectscontained in an image. This dramatically improves the ability of thesystem to quickly and accurately identify desired images over themethods currently available in the technology and industry.

Referring now to the flowchart of FIG. 1, a method of image comparisonaccording to one embodiment of the method begins at block 12, where astarting or query image is selected. The query image will typically beprovided by a user of the system and will comprise an image whichcontains one or more structures or objects of interest. Initially, thestructure of interest in the image may not be well defined or distinctrelative to the background. For example, the object boundaries may bepoorly delineated, or it may have significant internal features presentthat are not immediately apparent in the image.

To help define the object of interest, both in terms of its boundariesand its internal features, the system performs image filtering at block14. In advantageous embodiments, the filtering performed is under thecontrol of the system user. The system may also perform filteringautomatically using default filter functions or filter functionspreviously defined and stored by a user. A wide variety of well knownimage filtering techniques may be made available to the user. Many imagefiltering techniques which may be used in embodiments of the inventionare described at pages 151-346 of The Image Processing Handbook, 2dEdition, John C. Russ, author, and published in 1995 by CRC Press, whichis hereby incorporated by reference into this application in itsentirety. Several filters which are utilized in one embodiment of theinvention are set forth below with reference to FIGS. 4-6. These filtersmay enhance edges, enhance the appearance of pixels in particularbrightness ranges, stretch contrast in selected pixel brightness ranges,reduce noise, or perform any of a wide variety of pixel processingfunctions. It will be appreciated that the filtering performed at block14 may comprise the sequential application of several individual pixelfiltering functions. Advantageously, filtering performed in block 14 canresult in the enhancement of features which are characteristic ofobjects of interest or objects within a certain class, etc., but whichdo not appear in other objects or in the image background.

Following the filtering of block 14, objects within the filtered imageare defined at block 16. Once again, this process may be performed underthe control of the user, or performed automatically by the system. Ingeneral, this process involves evaluating pixel values so as to classifythem as either an object pixel or a background pixel. As with thefiltering performed at block 14, the object definition process of block16 may be done using many well known techniques, some of which aredescribed at pages 347-405 of The Image Processing Handbook mentionedabove. Example object definition protocols provided in one embodiment ofthe invention are described in more detail with reference to FIG. 7.

Next, at block 18, each defined object is separately numericallycharacterized by a set of parameters which are calculated from the pixellocations and brightness values of each defined object. In general, thenumerical parameters are measures of the object's shape, size,brightness, texture, color, and other calculated characteristics.Preferably, the values present in the parameter sets are similar forobjects of the same type. Example parameters which may advantageously beused in embodiments of the invention are described below with referenceto FIG. 3.

Referring now to block 20, a template for comparison is defined by theuser. The template may be a single defined object, or may be a group orcluster of defined objects in a region of the image. At block 22,similarities between the template and other objects or sets of objectsare calculated. If the template is a single object, this may be done bycomparing the parameter set assigned to the template object with theparameter sets assigned to other objects. There are several well knownways of evaluating the similarity between two parameter vectors. Forexample, Euclidean or Minkowski line metrics may be used. If theparameter set is represented as a bit string, the Hamming distance maybe used as the similarity measure.

In certain embodiments of the invention, multi-dimensional non-binaryparameter sets are associated with the objects, and as stated above, acomparison may be performed between not only individual parameter setsbut also between parameter set groups associated with clusters of aplurality of objects. In this case, more complicated formulae have beendeveloped and may be used, based on ideas set forth in Voronin, Yu. A.,Theory of Classification and Its Applications 1985, published in Russiaby Nauka. These formulae are set forth fully below. As is also explainedbelow, if the template comprises a set of two or more objects, thecomparison involves not only a comparison of the objects themselves, butalso the spatial relationship between them. This method for numericestimation of spatial relations between objects was developed by theinventors.

It will be appreciated that accuracy in identifying similar objects isimproved when the filtering and object definition steps described aboveresult in the enhancement of object features which are associated withobjects of the desired class but not associated with objects not in thedesired class. These enhanced features will manifest themselves as anumerically discriminable part of the parameter set, and the parameterset may thus be utilized to differentiate objects in the desired classfrom objects outside the desired class. Such differentiation manifestedby the system using object border contour displays. The system may usedifferent colors of the object border contours—blue for objects touchingthe image edges, green—for allowed non-border objects, red—for objectsfiltered out by the system based on user set parameters intervals, andyellow—for template objects.

As one specific example, a query image may comprise a digital image ofan area of skin pigmentation. A physician may be interested inevaluating the likelihood that the pigmentation in the image is amelanoma. Using a method according to the present invention, the digitalimage is filtered and an image area associated with the pigmentation isdefined as an object within the image. Other images of skin pigmentationwhich are stored in an image database are also filtered and areas ofskin pigmentation are defined as objects, advantageously using the samefilters and object definition functions. These objects in the databaseare then also parameterized. The query parameter set is compared to theparameter sets associated with the database objects, and images of skinpigmentation which are similar are identified. Advantageously, thepigmentation area of the stored images have been previouslycharacterized (diagnosed) as being melanoma or not. If retrieved similarobject images are predominantly images of melanomas, the physician maybe alerted that the possibility of melanoma for the query image is high.As mentioned above, it is advantageous if the filtering and objectdefinition procedures enhance those aspects of skin pigmentation imageswhich are closely associated with the presence of a melanoma.Furthermore, the parameter set itself may be tailored to the class ofobjects being analyzed. This may be done by assigning different weightsto the different parameters of the parameter set during the comparison.For the melanoma example, a high weight may be assigned to parameterswhich are indicative of an irregular boundary or surface, while a lowerweight may be assigned to a parameter associated with the total area ofthe object.

A system which may be used in one embodiment of the invention isillustrated in FIG. 2. An image acquisition device 26 is used toinitially create images for storage in an image database 24 and/or forrouting to a query image selection module 28 of the system. The imageacquisition device may be a source of images of any type, includingphotographs, ultrasound images, X-ray or MRI images, a CRT display ortrace, or any other data source having an output, which is definable asa collection of digital values. The image acquisition device may, forexample, be a digital camera. The image acquisition device may producethe image directly. The system may also import previously created imagesfrom one or more imaging sources. The image acquisition device may be anexternal digital imaging source for such systems like PACS, RIS, LIS orthe Internet or Telnet, for example. Typically, of course, the imagedata array processed by the system could be a two-dimensional array ofpixels wherein each pixel is assigned an associated scalar or vectorvalue. It is also well known that a two-dimensional array of pixels maybe derived from a real 3D object that was represented by 2-dimensional“slices” or scans. For grey scale images, each pixel is associated witha brightness value, typically eight bits, defining a gray scale fromzero (black) to 255 (white). For color images, a three component vectorof data values may be associated with each pixel. The query imageselection module, may, under the control of a user, select a query imagefrom the image acquisition device, or may retrieve an image from theimage database 24.

The system also comprises a display 30 which provides a visual output ofone or more images to the user of the system. For example, the queryimage itself will typically be displayed to the user with the displaydevice 30. This display of the query image may further be performedafter image filtering by the filter module 32 and object definition bythe object definition module 34. If no filtering or object segmentationhas yet been implemented by the user with these modules, the unprocessedquery image will be displayed to the user.

With a user input device 36 such as a keyboard, touchpad, or mouse, theuser may control the filter module 32 so as to implement the filteringdescribed above with reference to block 14 of FIG. 1. It is one aspectof some embodiments of the invention that the image continues to bedisplayed as the filtering is implemented. Thus, as the user modifiesthe filter function being performed by the filter module 32, the visualimpact of the filter application on the image is displayed to the user.

The user may also control the implementation of object definition by theobject definition module 34. Pixel brightness thresholds and otherfeatures of the object definition procedure may be modified by the userwith the input device 36. As with the filtering operation, the image maybe displayed after object definition so that the user can observevisually the contours and internal features of objects defined in theimage. If the object definition technique is modified by the user, thedisplay of the image may be accordingly updated so that the user canevaluate the effects of the filtering alterations and image objectchanges graphically on the display.

In some embodiments, the user may allow the system to perform objectdefinition automatically, without requiring any additional user input.Of course, the above described display updates may be performed afterthis automatic object definition as well. As is also illustrated in thisFigure and is explained further below with reference to FIG. 4, the usermay also control aspects of parameter calculation via the user inputdevice 36.

It will also be appreciated that in many applications, multiple imageshaving similar sources and structures will be processed by the user inthe same way (“batch processing”). For example, cranial X-ray images mayall be processed with the same filter set and object definitionfunctions prior to parameterization—in batch. This helps ensure thatcompatible images and objects therein are parameterized for comparison.Of course, care must be taken that the sources of the images arethemselves compatible. Overall brightness, dimensional variations, andother differences between, for example, different microscopes used toobtain the query image and images in the database 24 should becompensated for either prior to or as part of the processing procedures,known as dimension and/or brightness calibration.

To facilitate this common processing of multiple images user definedmacros of filter and object definition and detection functions may bestored in a macro database 35 for future use on additional images. Theuser-friendliness of the system is improved by this feature becauseimages from similar sources can be processed in the same way withoutrequiring the user to remember and manually re-select the same set offiltering and object definition functions when processing similar imagesin the future. In one embodiment, the user may operate on an image usingeither individual filter and object definition functions stored in themacro database or user defined groups of individual filter and objectdefinition functions stored in the macro database 35.

The object definition module 34 is connected to an objectparameterization module 38, which receives the pixel values and contourcoordinates of the objects defined in the image. This module thencalculates the parameter sets described above with reference to block 18of FIG. 1 using the input pixel values. The calculated parameter setsmay be stored in an index database 40 for future use. During the imagesearching, evaluating and retrieval process, one or more parameter setsassociated with a template will be forwarded to a parameter setcomparison module 42 along with parameter sets associated with otherobjects in the image or other objects in images stored in the imagedatabase 24. Objects or object clusters that are similar to thetemplate, are then also displayed to the user on the display 30.

Referring now to FIG. 3, it is one aspect of the invention that anygiven image may have associated with it several different parametersets, with each parameter set associated with a detected object in thatimage. Thus, the image database 24 may store a plurality of images 46,48, each of which includes a plurality of defined objects 50 a-d and 52a-b. Each object is associated with a parameter set 54 a-f, which isstored in the index database 40.

In one embodiment, the parameter set includes a computation of theobject area by a formula which counts the number of pixels defined aspart of object “A” and multiplies that number by a calibrationcoefficient as follows:

$\begin{matrix}{{\sum\limits_{i,j}\;{z*\delta_{i\; j}}},{\delta_{i\; j} = \{ {\begin{matrix}{1,{{i\; j} \in A}} \\{0,{{i\; j} \notin A}}\end{matrix},} }} & (1)\end{matrix}$

where z is a user defined dimensional calibration coefficient.

When the object has many internal holes, the area parameter may becalculated instead by the formula:

$\begin{matrix}{\frac{\sum\limits_{i}\;{( {X_{i} + X_{i - 1}} )*( {Y_{i} - Y_{i - 1}} )}}{2},} & (2)\end{matrix}$

wherein X, Y are the coordinates of the periphery pixels of the object.

Other advantageous object characterization parameters include the lengthof the perimeter, and the maximum and minimum diameters of the objectthrough the center of gravity of the object. These may be calculatedwith the formulas:

$\begin{matrix}{\sum\limits_{i}\;\sqrt{( {X_{i} - X_{i - 1}} )^{2} + ( {Y_{i} - Y_{i - 1}} )^{2}}} & (3)\end{matrix}$

for perimeter,

$\begin{matrix}{4*\sqrt{\frac{\begin{matrix}{\overset{\_}{x^{2}} - {\overset{\_}{(x)}}^{2} + \overset{\_}{y^{2}} - {\overset{\_}{(y)}}^{2} +} \\\sqrt{( {\overset{\_}{x^{2}} - {\overset{\_}{(x)}}^{2} - \overset{\_}{y^{2}} + {\overset{\_}{(y)}}^{2}} )^{2} + {4^{*}( {\overset{\_}{x\; y} - {{\overset{\_}{x}}^{*}\overset{\_}{y}}} )^{2}}}\end{matrix}}{2},}} & (4)\end{matrix}$

for maximum diameter, and

$\begin{matrix}{4*\sqrt{\frac{\begin{matrix}{\overset{\_}{x^{2}} - {\overset{\_}{(x)}}^{2} + \overset{\_}{y^{2}} - {\overset{\_}{(y)}}^{2} -} \\\sqrt{( {\overset{\_}{x^{2}} - {\overset{\_}{(x)}}^{2} - \overset{\_}{y^{2}} + {\overset{\_}{(y)}}^{2}} )^{2} + {4^{*}( {\overset{\_}{x\; y} - {{\overset{\_}{x}}^{*}\overset{\_}{y}}} )^{2}}}\end{matrix}}{2},}} & (5)\end{matrix}$

for minimum diameter, where

$\begin{matrix}{{\overset{\_}{x} = {( {\sum\limits_{j,{i \in A}}\; X_{i\; j}} )/( {\sum\limits_{j,{i \in A}}\;\delta_{i\; j}} )}},{\overset{\_}{y} = {( {\sum\limits_{j,{i \in A}}\; Y_{i\; j}} )/( {\sum\limits_{j,{i \in A}}\;\delta_{i\; j}} )}},} \\{{{\overset{\_}{x}}^{2} = {( {\sum\limits_{j,{i \in A}}\; X_{i\; j}^{2}} )/( {\sum\limits_{j,{i \in A}}\;\delta_{i\; j}} )}},{{\overset{\_}{y}}^{2} = {( {\sum\limits_{j,{i \in A}}\; Y_{i\; j}^{2}} )/( {\sum\limits_{j,{i \in A}}\;\delta_{i\; j}} )}},} \\{\overset{\_}{x\; y} = {( {\sum\limits_{j,{i \in A}}\;{X_{i\; j}^{*}Y_{i\; j}}} )/( {\sum\limits_{j,{i \in A}}\;\delta_{i\; j}} )}}\end{matrix}$

Other shape and size related parameters may be defined and included inthe parameter set, such as form factor:

$\begin{matrix}\frac{4*\pi*{Area}}{({Perimeter})^{2}} & (6)\end{matrix}$

equivalent circular diameter:

$\begin{matrix}\sqrt{\frac{4*{Area}}{\pi}} & (7)\end{matrix}$

and aspect ratio, which represents the ratio of the maximum diameter andminimum diameters through the center of gravity. The maximum and minimumFerret diameters of the object may also be included as part of theparameter set, namely:max X _(ij)−min X _(ij);max Y _(ij)−min Y _(ij),wherei,jεA  (8)

Parameters which relate to pixel intensities within the object are alsoadvantageous to include in the object characterization parameter set.These may include optical density, which may be calculated as:

$\begin{matrix}{- {\log_{10}( {\frac{\sum\limits_{{i\; j} \in A}\; I_{i\; j}}{\sum\limits_{{i\; j} \in A}\;\delta_{i\; j}}/I_{\max}} )}} & (9)\end{matrix}$

and integrated density:

$\begin{matrix}{\sum\limits_{i,{j \in A}}\; I_{i\; j}} & (10)\end{matrix}$

where I_(ij) is the brightness (i.e. 0-255 for 8-bit images or 0-65536for 16-bit images or 0-16777216 for 24-bit images) of pixel ij, andI_(max) is the maximum pixel brightness in the area/image.

More complicated intensity functions which parameterize the texture ofthe object may be utilized as well. One such parameter is a reliefparameter which may be calculated as:

$\begin{matrix}{{{\sum\limits_{i,{{i \in A};{{Nij} \geq 2}}}{{rl}_{ij}/{\sum\limits_{i,{{j \in A};{{Nij} \geq}}}\delta_{ij}}}},\text{where}}{{{rl}_{ij} = {r_{ij}*{\Omega({Nij})}}};}\text{where}{{\Omega( N_{ij} )}\mspace{11mu}{is}\mspace{14mu} a\mspace{14mu}{function}\mspace{14mu}{of}\mspace{14mu} N_{ij}}{{r_{ij} = {( {\sum\limits_{m = {i - 1}}^{i + 1}{\sum\limits_{n = {j - 1}}^{j + 1}{{abs}( {I_{nm} - I_{ij}} )}}} )/N_{ij}}};}{n,{{m \in A};}}{N_{ij} = {\sum\limits_{n = {i - 1}}^{i + 1}{\sum\limits_{m = {j - 1}}^{j + 1}\delta_{nm}}}}} & (11)\end{matrix}$

This parameter belongs to a textural class of parameters and is ameasure of the average difference between a pixel values in the objectand the values of its surrounding pixels. In the simplest case,Ω(N_(ij))=N_(ij), although the function may comprise multiplication by aconstant, or may involve a more complicated function of the number ofnearest neighbors or pixel position within the object.

Other examples include homogeneity:

$\begin{matrix}{{\Phi = {\sum\limits_{Ii}{\sum\limits_{Ij}( {N_{ij}/{\overset{\_}{N}( {DiameterFerret}_{xy} )}} )^{2}}}},} & (12)\end{matrix}$

where I is intensity; i, j ε A; and N is a renormalizing constant andcontrast:

$\begin{matrix}{{L = {\sum\limits_{{{Ii} - {Ij}} = 0}{( {I_{i} - I_{j}} )^{2}\lbrack {\sum\limits_{{Ii} - {Ij}}( {N_{ij}/{\overset{\_}{N}( {DiameterFerret}_{xy} )}} )} \rbrack}}},} & (13)\end{matrix}$

where I is intensity; i, j ε A; and N is a renormalizing constant

It will be appreciated that the nature of the parameter set may varywidely for different embodiments of the invention, and may includealternative or additional parameters not described above. The parametersset forth above, however, have been found suitable for objectcharacterization in many useful applications.

FIG. 4 illustrates a flowchart of the parameter set generation processwhich may be performed by the object paramterization module 38 of FIG.2. Initially, at block 55, the base or fundamental parameters arecalculated. These are the parameters that use raw pixel positions orintensities as inputs. Examples include area (Equation 1), perimeter(Equation 3), integrated intensity (Equation 10), etc. Another set ofparameters, referred to herein as “secondary” parameters are alsocalculated. These are parameters which are functions of the baseparameters, and which do not require any additional pixel specificinformation for their calculation. Examples of standard secondaryparameters include Formfactor (Equation 6) and aspect ratio. In someembodiments, the user is allowed to define additional secondaryparameters for object characterization which may have significance incertain image analysis applications. For example, a new hypotheticalparameter comprising the ratio of Formfactor to Area may be defined andmade part of the object characterization parameter set. Thus, at block56, the system may receive user input (by entering information into adialog box with a mouse and/or keyboard, for example) regardingsecondary parameter definitions not already utilized by the system.

At block 57 the system calculates both the user defined and standardsecondary parameters, and at block 58 the parameters thus calculated areformatted into a feature vector and output to either or both the indexdatabase 40 and the comparison and statistics system 42 of FIG. 2.

In FIGS. 5 through 9, a specific implementation of the invention isillustrated by example screen displays which illustrate aspects of usercontrol (via the input devices 36 of FIG. 2) and visualization (via thedisplay 30 of FIG. 2) of the filtering and object definition processes.As will be apparent to those of skill in the art, this embodiment of theinvention is implemented in software on a general purpose computer. Awide variety of data processing system environments may be utilized inconjunction with the present invention. In many embodiments, theinvention is implemented in software coded in C/C++ programminglanguages and running on a Pentium series personal computer with, forexample, as little as 128 Mbytes of RAM and a 640 MB hard drive. Thepersonal computer in this implementation will typically be connected toan image database through a local or wide area network, or via PACS,RIS, LIS or Internet/Telnet client-server system. In anotherimplementation, the personal computer runs a standard web browser, whichdisplay a communicating application and accesses image databases andimage analysis and computer-aided detection software hosted on a remoteInternet server. Intranet version of the application is also envisionedand implemented. In such case the system works as a part of PACS, forexample, using LAN and HIS as a hosting system.

Referring now to FIG. 5, original images 60 a and 60 b are displayed tothe user of the system in respective portions of the display. The upperdisplay 60 a comprises a close up of a suspected malignancy in amammogram. The lower display 60 b is a bone density image utilized inevaluating osteoporosis. On another portion 62 of the screen is adisplay of a filter protocol. This portion 62 of the screen displayshown one of the computationally simplest filtering techniques underuser control in this embodiment, which is look-up-table (LUT) filtering.With this filter, each input pixel brightness value is mapped onto anoutput pixel brightness value. If pixel brightness ranges from a valueof 0 (black) to 255 (white), each value from 0 to 255 is mapped to a newvalue defined by the LUT being used.

In this embodiment, the user is provided with a visual indication 64 ofthe look-up table form being applied, with input pixel values on thehorizontal axis and output pixel values on the vertical axis. Using userselectable check boxes 63, the user may define the nature of thelook-up-table filter being applied. In this embodiment, the user maydefine both a table form and a table function. The form may be selectedbetween linear (no effect on pixel values), triangular, and sawtooth(also referred to as notch). The triangular form is illustrated in FIG.5. For the triangular and sawtooth forms, the user may be provided witha slidebar 66 or other input method for selecting the number of periodsin the input brightness range. The user may also import a previouslyused user defined LUT if desired.

The look-up-table form may also be varied by additional user definedfunctions. These functions may include negative inversion,multiplication or division by a constant, binarization, brightnessshifting, contrast stretching, and the like. For each of thesefunctions, the user may control via slidebars or other usermanipulatable displays the constants and thresholds utilized by thesystem for these functions. Histogram based look-up table filtering mayalso be provided, such as histogram equalization and histogram basedpiecewise contrast stretching. After the user defines the desired LUTfilter, they may apply it to the image by selecting the “APPLY” button68. The look-up-table defined by the user is then applied to the imageor a selected portion thereof.

Furthermore, second display 70 a and 70 b of the image is providedfollowing application of the three period triangular LUT filter. If theuser modifies the LUT filter function, the image display 70 a, 70 b isupdated to show the visual result of the new filter function when theuser clicks the APPLY button 68. Thus, the user may view a substantiallycontinuously updated filtered image as the filter functions used aremodified. In filtered image 70 a, regions of suspected malignancy areenhanced with respect to the background following LUT application. Inthe filtered image 70 b, the bone density variations present in thecentral bone segment are enhanced and pronounced.

In addition to LUT filtering, convolution filters, frequency domainfilters, and other filter types may be utilized to further enhance anddefine significant features of imaged objects. Several specific examplesprovided in one embodiment of the invention are illustrated in FIGS. 6and 7. In analogy with the user interface for the LUT filteringdescribed with reference to FIG. 5, additional filter types may beselected with checkboxes 78, 80. Filter parameters such as filter boxsize are user controllable via slidebars 82, 84. APPLY buttons 86, 88initiate the filter operation and display update to show the filteredimage or image region. In FIG. 6, the bone image 60 b is filtered with a3×3 edge detection filter which produces the filtered image 87 havingenhanced pixels along edges in the image. In FIG. 7, a region ofinterest 89 in an image of blood cells in bodily fluids where a shadingfilter was used to compensate for a background brightness variationacross the image.

In the specific implementation illustrated in FIGS. 6 and 7, thefollowing base set filter functions may be applied by the system user:

1. Sharpening of Small Size Details on Image

This type of filter belongs to a class of Laplacian filters. The filteris a linear filter in the frequency domain. The 3×3 kernel is understoodto mean that central pixel brightness value is multiplied by 4. As aresult of this filtering, the sharpness of small details (not to exceed3×3) of the image is increased.

$C_{mn} = \begin{Bmatrix}{- 1} & {- 1} & {- 1} \\{- 1} & 9 & {- 1} \\{- 1} & {- 1} & {- 1}\end{Bmatrix}$

2. Sharpening of Middle Size Details on Image

This type of filter belongs to a class of Laplacian filters.Functionality is similar to the 3×3 kernel type filter. As a result ofthis filtering, the sharpness of small details (not to exceed 5×5) ofthe image is increased.

$C_{mn} = \begin{Bmatrix}{{- 1}/12} & {{- 1}/12} & {{- 2}/12} & {{- 1}/12} & {{- 1}/12} \\{{- 1}/12} & {{- 2}/12} & {3/12} & {{- 2}/12} & {{- 1}/12} \\{{- 2}/12} & {3/12} & {28/12} & {3/12} & {{- 2}/12} \\{{- 1}/12} & {{- 2}/12} & {3/12} & {{- 2}/12} & {{- 1}/12} \\{{- 1}/12} & {{- 1}/12} & {{- 2}/12} & {{- 1}/12} & {{- 1}/12}\end{Bmatrix}$

3. Sharpening of a Defined Size Details on Image

This filter performs convolution transformation of the image through auser defined multiplication factor. As a result, all details of a userdefined size are sharpened. The size of processed image detail may bedefined through available editing submenu windows for X and Ydimensions.

$\begin{matrix}{{{I_{out} = {I_{in}*\vartheta*( {I_{m} - {\sum\limits_{\Omega}{I_{in}/( {m*n} )}}} )}},{{where}\mspace{14mu}\vartheta\mspace{14mu}{is}\mspace{14mu}{the}\mspace{14mu}{user}\mspace{14mu}{defined}\mspace{14mu}{multiplication}\mspace{14mu}{factor}}}{{and}\mspace{14mu}\Omega\mspace{14mu}{is}\mspace{14mu}{the}\mspace{14mu}{mxn}\mspace{14mu}{filter}\mspace{14mu}{box}}} & (14)\end{matrix}$

4. Sharpening of a Low Contrast Details

This filter performs convolution transformation of the image and belongsto a spatial domain filters. The filtering is performed through a userdefined multiplication Factor and automatically calculated specialparameter. This parameter is a ratio of a current pixel value to MeanSquare Deviation of a pixel value calculated for the given size of thepixel aperture (or filter box). As a result, all details of a userdefined size are sharpened. The size of the processed image detail maybe defined through available for editing submenu windows for X and Ydimensions.

$\begin{matrix}{{I_{out} = {I_{in}*\vartheta*\mu*( {I_{in} - {\sum\limits_{\Omega}{I_{in}/( {m*n} )}}} )}},{{where}\mspace{14mu}\vartheta\mspace{14mu}{is}\mspace{14mu}{factor}\mspace{14mu}{and}\mspace{14mu}\mu\mspace{14mu}{{{is}( {\sum\limits_{\Omega}{I_{in}/( {m*n} )}} )}/\sigma_{\Omega}}}} & (15)\end{matrix}$

5. Edge Enhancement Filter

This edge enhancement filter belongs to a non-linear range filter. Userdefines the size of the filter box. This filter provides two regimes,selected by the user, If the default regime Strong is changed by theuser to regime Weak, the filter will change the processing method toavoid images noise impact in certain high frequencies.I _(out) =Sup _(Ω),when I _(in)>1/2*(Sup _(Ω) +I _(Ω))I _(out) =Inf _(Ω),when I _(in),≦1/2*(Sup _(Ω) +Inf _(Ω))  (16)

where Sup_(Ω) is maximum brightnesss within filter box and Inf_(Ω) isminimum brightness within filter box

6. Edge Detection

This edge detection filter belongs to modified Laplacian omnidirectionaledge detection convolution filters. User defines the size of the filterbox. This filter performs edge detection of the image through a userdefined Factor. The Factor is used for convolution mask valuescalculations

7. Dilation Filters

Both filters belong to morphological class and are inversive to eachother. The first one should be used for image light elements dilation,the second one—for dark elements dilation. If the default regime Strongis changed by the user to regime Weak, both filters will change theprocessing method to avoid images noise impact in certain highfrequencies. In general:I _(out) =Sup _(Ω) or I _(out) =Inf _(Ω)  (17)

8. Low Frequency

This filter represents a convolution transformation of modified Gaussiantype. It belongs to a class of linear filters in frequency domain. Thesize of pixel box or aperture is defined by the user for X and Ydimensions. The filter is used often for certain frequencies noisereduction. In general:

$\begin{matrix}{I_{out} = ( {\sum\limits_{\Omega}{I_{in}/( {m*n} )}} )} & (18)\end{matrix}$

9. Gradient/Modified Sobel Edge Detection Filter

This filter belongs to a non-linear edge-detection class. The filteruses a technique with partial derivatives replacement with theirestimates. It is known in image processing as a Sobel filter. The sizeof the pixel box or aperture defined by the user for X and Y dimensions.This filter performs convolution transformation of the image through auser defined amplification Factor. The user also is provided with theability to set a binarization Threshold if a correspondent check-box ismarked. The threshold serves as a modification to the classic Sobelfilter and enables the user to find right flexibility for the edgedetection process. If the threshold is used the outcome oftransformation will be a binary image. The default but modifiable masksare:

$C_{mn} = {{\begin{Bmatrix}1 & 0 & {- 1} \\2 & 0 & {- 1} \\1 & 0 & {- 1}\end{Bmatrix}\mspace{50mu} C_{mn}} = \begin{Bmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\1 & 2 & 1\end{Bmatrix}}$

10. Shading Correction

This filter belongs to a smoothing class filter. The size of the pixelbox or aperture is defined by the user for X and Y dimensions. Thefilter is modified from a classical type shading correction filter byenabling the user with shifting capability. If check-box Shift is markedthe user will be able to change the default value of the shift to acustom one. This filter is very handy for elimination of a negativelighting impact which sometimes occurs during the image acquisitionprocess.

$\begin{matrix}{{I_{out} = {( {I_{in} - {\sum\limits_{\Omega}{I_{in}/( {m*n} )}}} ) + {Shift}}},{{whereShift}\mspace{14mu}{by}\mspace{14mu}{default}\mspace{14mu}{is}\mspace{14mu} 127}} & (19)\end{matrix}$

11. General or Universal Filter

This is a convolution type filter with a user controlled size of thekernel and the weights mask values. The default size of the kernel is9×9. For the user's convenience, the convolution mask contains defaulttypically used weights values. Push-button activates the customizationregime when the user is able to modify dimensions of the mask and thenmodify default weights in the convolution mask.

12. Median (3×3) Filter

Moving median (or sometimes referred as rank) filter produces as anoutput the median, replacing a pixel (rather than the mean), of thepixel values in a square pixel box centered around that pixel. Thefilter is a non-linear type filter with the filtration window dimensionsof 3×3. Usually used to eliminate very small details of the image sizedat 1-2 pixels.

13. Median (5×5) Filter

Similar to the filter described above, but with the filtration windowdimensions 5×5. Usually used to eliminate small details of the imagesized at up to 5 pixels.

14. General Median Filter

This filter is similar to the filters described above, but with thefiltration window dimensions set by the user. The size of eliminateddetails depend on the size of the set filtration window.

15. Psuedomedian Filter

This filter is similar to median type filters described above. Howeverit provides rectangular filtration window controlled by the user andperforms transformation in a two pass algorithm.

User control of object definition (corresponding to module 34 of FIG. 2)is illustrated in FIG. 8. By selecting one of the checkboxes 92, theuser implements manual or semi-automatic object definition. In manualmode, slidebars allow the user to select a brightness range of pixels.All pixels outside this range are considered background. An object isthus defined as a connected set of pixels having brightness values inthe user defined range. Background pixels may be reassigned a zerobrightness value. In the automatic mode, the user interface for which isillustrated in FIG. 8, the thresholds are calculated automatically bythe system from the image histogram. In this mode, the system may allowthe user to set up multiple thresholds by setting their values manuallyor by choosing their sequential numbers from the automaticallycalculated table of thresholds.

As was the case with the filtering process, the image (or region ofinterest) is displayed as the object definition function is applied.Those of skill in the art will understand that a wide variety oftechniques for assigning pixels to objects or background are known andused, any one of which now known or developed in the future may be usedin conjunction with the present invention.

After objects are defined/detected, parameter sets are calculated foreach object, and then comparisons are possible to find similar objects(or object clusters as discussed above) in either the same image or indifferent images. This is illustrated in FIG. 9, which shows a displayof the original image 104 after filtering and object segmentation, aswell as the template 106 selected for comparison to objects in theremainder of the image. In this example, the template 106 is a threeobject cluster. Also provided in this screen display are seven displays108 a-g which display in rank order the seven objects of the image mostsimilar to the template object. Also displayed at 110 is a list of theparameters used in the comparison and the weights assigned to them forthe comparison process. These weights may be manually set, or they maybe set via a statistical process which is described in further detailbelow.

The actual comparison process which defines the degree of templatesimilarity may, for example, be performed with the following formulas.For templates consisting of one individual parameterized object, aparameter difference vector may be computed which has as each elementthe difference between the parameter values divided by the maximumdifference observed between the template object and all objects beingcompared to the template.Δ_(it)(P _(it) ,P _(j))/Δ_(max)(P _(it) ,P _(k)),  (20)whereP is a parameter-vector; _(it) is the index of template object; _(k)=1,. . . , L; L is all objects that the template object is being comparedto; and _(j) is the index of specific object being compared to templateobject.

A numerical similarity may then be computed using either a modified formof Euclidean or Minkowski line metrics or as modified Voronin formula asset forth below:

$\begin{matrix}\{ {{\begin{matrix}( {\sum\limits_{k = 1}^{L}{( {p_{k}^{t} - P_{k}^{t}} )^{s}*{\omega\;}_{k}}} )^{1/s} \\\text{and} \\{{( {P_{i} - P_{k}} )^{T}{W^{- 1}( {P_{i} - P_{k}} )}},} \\{{\text{where}\mspace{14mu} W\mspace{14mu}\text{is}\mspace{14mu}\text{the}\mspace{14mu}\text{covariation}\mspace{14mu}\text{matrix}};} \\{\omega\mspace{14mu}\text{is}\mspace{14mu}\text{a}\mspace{14mu}\text{statistical}\mspace{14mu}\text{weight}}\end{matrix}\text{and}\mspace{14mu}{in}\mspace{14mu}\text{our}\mspace{14mu}\text{modification}\mspace{20mu}\text{is}p} = {p_{k}^{t}/( {{\max\; p_{k}} - {\min\; p_{k}}} )}}  & (21)\end{matrix}$

For multi-object templates or entire images, the spatial relationshipbetween selected objects of the template to other objects in thetemplate may be numerically characterized and effectively added as oneor more additional subvectors of the object parameter vector. Theoverall similarity between a multi-object template and object clustersin the image database, may, in some embodiments of the invention becalculated as follows:

$\begin{matrix}{{{\zeta = {\sum\limits_{j = 1}^{Z}{\varpi*{{{abs}( \eta_{ij}^{t} )}/Z}}}},\text{where}}\mspace{14mu}{{Z - {\text{number}\mspace{14mu}\text{of}\mspace{14mu}\text{components}}},{\eta_{ij}^{t} = {1 - {{{abs}( {\Delta_{i}^{t} - \Delta_{j}^{t}} )}/( {{\max\;\Delta_{t}} - {\min\;\Delta_{t}}} )}}},{\Delta^{t} = \{ \begin{matrix}{1,} & {{\text{when}\mspace{14mu}{{abs}( {\Delta_{i}^{t} - \Delta_{j}^{t}} )}} \leq ɛ_{t}} \\{0,} & \text{else}\end{matrix} }}} & (22)\end{matrix}$ε is a thresholds and/or tolerances vector,

{tilde over (ω)} is a weights vector

This formula combines not only parametric similarity but spatialsimilarity also. For spatial similarity the closeness of the positionand pattern fit for objects of the template and objects of the databaseare numerically evaluated. The mathematical method for parameterizingthese spatial relationships may, for example, use some simple Euclideandistances between objects for primitive cases and up to pattern fitcalculations based on second, third, or fourth moments of inertia forcomparable components in complex cases.

Once the objects are parameterized and the template is defined as eithera single object or a cluster of objects, the comparison calculationinvolves the mathematical generation of a value which characterizes how“similar” two vectors or matrices of numbers without further referenceto the meaning associated with those numbers. A wide variety ofmathematical techniques are available to perform such a numericalcharacterization, and different approaches may be more suitable thanothers in different contexts. Thus, the specific formalism used tomathematically define and quantify similarity between number sets mayvary widely in different embodiments of the invention and differenttechniques may be appropriate depending on the application.

As discussed above, the weight assigned to a given parameter during thiscomparison process may be manually set by the user or set using astatistical method. The statistical method is especially useful when thedatabase of images includes a large number of objects which have beencharacterized as having or not having a characteristic trait, such as anarea of skin pigmentation is either melanoma or not melanoma, or whichhave been characterized numerically as more similar or less similar to a“model” object. When this data is available, it can be analyzed todetermine how strongly different parameters of the parameter set valuescorrelate with the presence or absence of the specific trait.

The weight used for a given parameter in the comparison process may thusbe derived from the values of the parameter vectors associated with thedetected objects in the image database.

In using this method a system is represented as a totality of factors.The mathematical simulation tools are correlation, regression, andmultifactor analyses, where the coefficients of pairwise and multiplecorrelation are computed and a linear or non-linear regression isobtained. The data for a specific model experiment are represented as amatrix whose columns stand for factors describing the system and therows for the experiments (values of these factors).

The factor Y, for which the regression is obtained, is referred to asthe system response. (Responses are integral indicators buttheoretically, any factor can be a response. All the factors describingthe system can be successively analyzed.). The coefficients of theregression equation and the covariances help to “redistribute” themultiple determination coefficient among the factors; in other words the“impact” of every factor to response variations is determined. Thespecific impact indicator of the factor is the fraction to which aresponse depending on a totality of factors in the model changes due tothis factor. This specific impact indicator may then be used as theappropriate weight to assign to that factor (i.e. parameter of theparameter set associated with the objects).

The impact of a specific factor is described by a specific impactindicator which is computed by the following algorithm:γ_(j) =α*[b _(j) *c _(0j) ], j=1,2, . . . ,k  (23)

where γ is the specific impact indicator of the j-th factor; k is thenumber of factors studied simultaneously; bj is the j-th multipleregression coefficient which is computed by the formulaX ₀ =a+Σb _(j) *Xj,  (24)

where X₀ is the system response to be investigated, a is a free term ofthe regression, and X_(j) is the value of the j-th factor. Thecoefficient α of the equation is computed by the formulaαR ² ,[Σ _(j) |b _(j) *c _(0j)|]  (25)

where R is the coefficient of multiple determination computed by theformulaR=[(n ²*Σjb _(j) *c _(0j))/(n*Σ _(j) x ² _(0j)−(Σ_(j) x_(0i))²)]^(1/2),  (26)

where n is the number of observations, which cannot be below (2*K);x_(0i) is the value of the system response in the i-th observation,c_(0j) is the covariance coefficient of the system response indicatorand the j-th factor. It is given by the relationc _(0j)=(n*Σ _(i) x _(0i) *x _(ji)−Σ_(i) x _(0i)*Σ_(i) x _(ji))/n²  (27)The specific contribution indicator is obtained mainly from thecoefficient of multiple determination, which is computed by the formulaR ²=(Σ_(j) b _(j) *c _(0j))/D ²  (28)where D² is the response variance. The specific impact of the j-thfactor on the determination coefficient depends only on the ratio ofaddends in this formula. This implies that the addend whose magnitude isthe largest is associated with the largest specific impact. Since theregression coefficients may have different signs, their magnitudes haveto be taken in the totals. For this reason, the coefficients γ of thespecific impact are bound to be positive. However, it is important thatthe direction in which the factor acts by the computed γ is dictated bythe sign of the regression coefficient. If this sign is positive, theimpact on the response variable is positive and if it is not, theincrease of the factor results in a reduction of the response function.The influence of the background factors, which are not represented inthe data, is computed by the formula{tilde over (γ)}_(i)=1−Σ_(j)γ_(j).  (29)The importance of the γ is determined from the relation for theempirical value of the Fisher criterionF _(j)=(γ_(j)*(n−k−1))/(1−Σ_(j)γ_(j)).  (30)

A rearrangement of the initial data matrix at every experimental stepmakes it possible to investigate successively the dynamics of thesignificance of the impact the factors have on all system indicatorsthat become responses successively. This method increases thestatistical significance of the results obtained from the algorithm forthe recomputation of the initial data matrix. The algorithm embodiesserial repeatability of the experiments by fixing the factors at certainlevels. If the experiment is passive, the rows of the initial matrix arechosen in a special way so that, in every computation, rows with theclosest values of factors (indicators) influencing the response aregrouped together. The dynamics of the specific contributions is computedby using the principle of data elimination.

In the proposed way, the computation of the dynamics of theinsignificant information is gradually eliminated. The value of γ doesnot change remarkably until the significant information is rejected. Adramatic reduction of γ is associated with a threshold with which thiselimination of useful information occurs. The algorithm of thisoperation is an iterative γ recomputation by formula (23) and arejection of information exceeding the threshold computed. In thealgorithm, the significance of the result and of the informationeliminated is increased by recomputing the initial data matrix into aseries-averaged matrix, the series being, for instance, the totality ofmatrix rows grouped around the closest values of the factor in the caseof a passive factorial experiment. The series may also consist ofrepeated changes of the indicator with the others fixed at a specifiedlevel. Because in further discussion the series-averaged matrix isprocessed in order to obtain final results, the compilation of seriesfrom the data in a field is a major task for the user because, both, thenumerical and meaningful (qualitative) result of the computation may beinfluenced. With increasing threshold the amount of rejected informationalso increases, therefore one has to check whether the amount ofinformation in the series-averaged matrix is sufficient, see below.Consequently, the information on the factor considered in this versionof the method is rejected by the formulaX _(1i)=[Σ_(p) X _(1ip) −m*h]/n _(i) , p=1,2, . . . m; i=1,2, . . . ,N,where X_(1i) is the value of the i-th series in which the factor X₁ isobserved and for which the critical (rejection) threshold is determinedafter the elimination of data with a threshold of H; n_(i) is the numberof observations in the i-th series; m is the number of values of the X₁which exceed h and (0≦m≦n_(i)); N is the number of observation series(rows of the N*(K+1) matrix of the initial information, where K is thenumber of factors investigated simultaneously.)

The invention thus provides image searching and comparison based in amuch more direct way on image content and meaning than has beenpreviously available. In addition, using the described method of weightscalculations for targeting similarities between a multi-componenttemplate and a database of images in medical fields is much moremathematically justified and sound than neural network techniques usedfor the same purposes. That is important to understand because templatematching may be used in such applications to decrease the difficulty ofdatabase creation and search, and improve early cancer diagnostics,early melanoma detection, etc.

As set forth above, diagnosis or estimation of level of likelihood ofpotential disease states is facilitated by noting that an object in aquery image is or is not similar to objects previously classified asactual examples of the disease state. In some embodiments, diagnosis orlevel of likelihood of potential disease states is facilitated bycomputing a numerical score which is indicative of the likelihood that aparticular diagnosis (e.g. malignant melanoma or benign growth, benignbreast lesion or carcinoma) is correct. This score may be computed by ananalysis of the numerical similarity scores between an object or objectsin the query image and previously classified objects in the database.Several new methods are proposed as set forth below.

Algorithm 1: This is a first order ranking method, essentially a binaryclassification of the query object. The software calculates andretrieves the T_(ψ) closest matches in the database to the unknownobject. The database objects were previously detected, defined andquantified. Then the rank is assigned according to a rule: if more thana half of the closest template objects T_(ψ) have been diagnosed as nodisease then the score for the unknown object shall reflect no diseasefinding, otherwise the score reflects disease or its likelihood.

Algorithm 2. This is a simple Averaging Ranking Scoring system.Continuum similarity values for the closest T_(ψ) templates objects withknown findings are substituted by their dichotomic ranks (e.g.—1 forbenign or 5 for malignant, or 1 for presence of the disease and 0—forits absence). Then the assigned score is an average of the T_(ψ) ranks.

Algorithm 3. Scoring with the penalty function. The method uses only themaximum number ΓΩ of closest templates objects that corresponds to thehighest ranking value τ_(max) in the scoring range. The values ofcalculated similarities between each template with known finding and theunknown object is substituted with the values that are calculated asfollows:

For Templates of highest τ_(max):τ_(max)−Penalty*Relative Similarity;

For Templates of τ_(min):τ_(min)+Penalty*Relative Similarity.

For example, if τ_(max) is equal 5 and τ_(min) is equal 1 and theRelative Similarity based retrieved closest matches for cluster of 6 are(62.24% 60.78% 60.48% 59.68% 59.49% 59.23%) with diagnostic findings asfollows (benign malignant benign benign benign benign maligant) then thescore for. i.e. second template in the cluster will be equal to5+(5−1)*(60.78−100)/100=3.431.

Algorithm 4. Averaging with weights for position with fixed retrievedtemplates cluster method. The software calculates and retrieves theΓ_(ψ) closest matches to the unknown object that represents themanifestation of the disease (i.e. lesion, skin growth, etc). Theseobjects were detected, defined and quantified. Continuum similarityvalues for the closest Γ_(ψ) templates objects with known findings aresubstituted by their dichotomic ranks (i.e. —1 for benign or 5 formalignant, or 1 for presence of the disease and 0—for its absence). Thenthe assigned score is an average of the Γ_(ψ) ranks, however each rankis multiplied by the evenly distributed weight calculated for itsposition in retrieved cluster. Each weight can be calculated indifferent ways—for example as follows: for each position above themiddle position of the cluster the current rank gets its weightincreased by 1, for every position below the middle position of thecluster the current rank gets its weight decreased by 1 (i.e. if thecluster N_(c) is 7 then the score of the closest Γ_(ψ) template objectwill have its weight of (7+1+1+1)/7=10/7. In other words if we have thefollowing sequence of the closest matchesmalignant-benign-benign-malignant-malignant-benign-malignant in N_(c)=7templates cluster and malignant is indicated by the score 5 and benignis indicated by the score 2 then the calculated total score will be(5*10/7+2*9/7+2*8/7+5*7/7+5*6/7+2*5/7+5*4/7)17=3.653).

Algorithm 5. Averaging with weights for position method with floatingretrieved templates cluster method. The method is similar to Algorithm 4except number N_(c) of templates in each retrieved cluster is truncated.The truncation could be done by setting Relative Similarity thresholdto, say, 80% or 90%. This way all templates with Relative Similaritybelow the threshold will not be considered and the value of N_(c) willnot be constant like in Algorithm 4.

In the example of FIG. 10, existing multiple slices of 3D ultrasoundimage of a breast lesion were processed by the system, segmented and theselected few scored against digital database of templates with knownfindings. The result of the database search, retrieval and scoring wasdisplayed in a form of 7 closest matches found and overall score isproduced (in our case 2—benign) by one of the five scoring methodsdescribed herein below. Then the system rendered 3D image of theprocessed lesion slices facilitating further quantification of thelesion such as analyses of volume, vortex as well as estimations of thetexture and curvature of the lesion surface. It is possible to compareand quantify relative similarity not only individual slices of thelesion but also the rendered 3D lesion or mass as a whole object.

The foregoing description details certain embodiments of the invention.It will be appreciated, however, that no matter how detailed theforegoing appears in text, the invention can be practiced in many ways.As is also stated above, it should be noted that the use of particularterminology when describing certain features or aspects of the inventionshould not be taken to imply that the terminology is being re-definedherein to be restricted to including any specific characteristics of thefeatures or aspects of the invention with which that terminology isassociated. The scope of the invention should therefore be construed inaccordance with the appended claims and any equivalents thereof

What is claimed is:
 1. A method for facilitating diagnosis or suspicionof a condition related to an X-ray, MRI, mammography, or ultrasound,comprising: processing user input to identify or define an object ofinterest in a first display image, the object of interest being locatedwithin the first display image and representing a candidate condition orindication for diagnosis or suspicion of the condition; displaying theobject of interest in a first display; determining at least one imagingparameter based on at least one measurement of the object of interest;displaying, near the first display, a plurality of additional imagesfrom an identified database, wherein each of the plurality of additionalimages comprises one or more objects and each of the plurality ofadditional images is identified based on mathematically generating avalue which characterizes a similarity measurement between the at leastone imaging parameter of the objects of interest and the at least onestored parameter of an object within the corresponding additionalimages; spatially arranging the plurality of additional images accordingto the similarity measurement and selecting, by a user, at least one ofa plurality of image sharpening options with checkboxes; outputting afirst result for a first analysis of the user input based on pixellocation; outputting a second result for a second analysis of the userinput based on the first result of the first analysis; and outputting anumeric score that indicates the likelihood that a disease is associatedwith the user input, wherein the similarity measurement is based on thefirst result and the second result, and the similarity measurement iscalculated with a modified Voronin formula.
 2. The method of claim 1,wherein each of the plurality of additional images is smaller than thefirst display image and wherein the first display image is displayedsimultaneously to the plurality of additional images.
 3. The method ofclaim 2, wherein each of the plurality of additional images is a portionof a full representation of an original referenced images.
 4. The methodof claim 1, wherein the mathematical generation of a value whichcharacterizes the similarity measurement fails to refer to the meaningassociated with the at least one imaging parameter.
 5. The method ofclaim 4, wherein the at least one imaging parameter comprises aparameter that is a numeric representation of at least one of geometry,texture and density of the first display image.
 6. The method of claim1, wherein: the object of interest comprises multiple components, and amulti-object template is used to determine the similarity measure basedon at least a spatial relationship between one or more objects of themulti-object template to other objects in the multi-object template. 7.The method of claim 1, wherein the identified database is a referencelibrary of cases with known or confirmed diagnosis or condition.
 8. Themethod of claim 1, further comprising selecting the imaging parametersbased on a pixel analysis.
 9. The method of claim 8, wherein the pixelanalysis is a pixel location analysis.
 10. The method of claim 8,wherein the pixel analysis is an image geometry, image texture, or imagedensity analysis.
 11. The method of claim 1, further comprisingdetermining at least one secondary parameter as a function of the atleast one imaging parameter, wherein each the plurality of additionalimages is identified based on both the at least one imaging parameterand the at least one secondary parameter.
 12. The method of claim 11,wherein the at least one secondary parameter is a risk factor, an age ofa patient, or a disease history of a patient.
 13. The method of claim 1,wherein the similarity measurement between the at least one imagingparameter of the objects of interest and the at least one storedparameter of an object is measured by Mikowski line metrics or measuredby Hamming distance.
 14. A method for facilitating diagnosis orsuspicion of a condition related to an X-ray, MRI, mammography, orultrasound, comprising: displaying a first image comprising an object toa user; and displaying, near the first image, a plurality of additionalimages from an image repository, wherein: each of the plurality ofadditional images comprises one or more objects, each of the pluralityof additional images is identified based on a mathematically generatingof a value which characterizes a similarity measurement between the atleast one stored parameter of an object within the correspondingadditional image and the at least one imaging parameter of an object ofinterest of the first image, each of the plurality of additional imagescorresponds to a known diagnosis, the plurality of additional images arespatially arranged according to the similarity measurement andselecting, by a user, at least one of a plurality of image sharpeningoptions with checkboxes; outputting a first result for a first analysisof the user input based on pixel location; outputting a second resultfor a second analysis of the user input based on the first result of thefirst analysis; and outputting a numeric score that indicates thelikelihood that a disease is associated with the user input, wherein thesimilarity measurement is based on the first result and the secondresult, and the similarity measurement is calculated with a modifiedVoronin formula.
 15. The method of claim 14, wherein the at least oneimaging parameter comprising a parameter selected from a groupconsisting of a numeric representation of geometry, texture and density.16. The method of claim 14, wherein each of the plurality of additionalimages is smaller than the first image and wherein the first image isdisplayed simultaneously to the plurality of additional images.
 17. Themethod of claim 14, wherein each of the plurality of additional imagesis a portion of a full representation of an original referenced imagewith known or confirmed diagnosis or condition.